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Why This Matters 


The Department of Education gives grants to schools 
and organizations that provide disadvantaged students 
with services to help them attend college. These eight 
grant programs are collectively known as “TRIO”, 
named for the original three programs. 


Congress provides over $1 billion each year to these 
programs, but Education could do more to understand 
how well these grants work to help students. 


Key Takeaways 


Education could improve the information it has about 
TRIO programs in two areas: (1) grantee performance 
data, and (2) program assessments. 


Schools and organizations report data to Education to 
show how the TRIO grants they receive have been 
working. For example, organizations that receive grants 
to encourage students to complete college report on the 
numbers and percentages of students who received 
services and earned degrees. 


Education evaluates grantees’ performance using the 
self-reported data, but has done little to verify the data. 
Accurate performance data are important because 
returning grantees can earn points for past performance 
in the next grant competition—increasing the likelihood 
that they will receive new grants. Almost 80 percent of 
recent TRIO grants went to returning grantees. 


Therefore, grantees may have an incentive to report a 
more positive picture than warranted. Officials from an 
organization representing TRIO grantees told us there is 
a risk that some grantees may report inaccurate 
information. 


As for assessing the individual TRIO programs, studies 
of some programs are outdated. In addition, Education 
has never assessed the effectiveness of three of the 
seven TRIO programs that serve students, and did not 
have any new assessments planned as of August 2020. 
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How GAO Did This Study 


We analyzed data from Education about TRIO grantees 
and applicants. We also reviewed relevant federal laws 
and regulations and agency documents, and interviewed 
Education officials and other TRIO stakeholders. 


What GAO Recommends 


Education should take additional steps to ensure the 
reliability of grantees’ performance data and develop a 
plan for assessing the effectiveness of the TRIO 
programs that serve students. Education generally 
agreed with our recommendations. 


For more information, contact: Melissa Emrey-Arras at (617) 
788-0534 or emreyarrasm@gao.gov 
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GAO U.S. GOVERNMENT ACCOUNTABILITY OFFICE 


441 G St. N.W. 
Washington, DC 20548 


December 17, 2020 


The Honorable Virginia Foxx 
Republican Leader 

Committee on Education and Labor 
House of Representatives 


Dear Dr. Foxx: 


In recent years, Congress has provided over $1 billion annually for eight 
college access grant programs known as the Federal TRIO programs 
(TRIO). These programs assist approximately 800,000 disadvantaged 
students with preparing for, enrolling in, and graduating from college. 
Administered by the Department of Education’s (Education) Office of 
Postsecondary Education, TRIO provides over 3,000 competitive grants 
to organizations across all 50 states.1 Grantees are primarily colleges, but 
school districts and community organizations also receive these grants. 
TRIO primarily serves low-income and first-generation students as well as 
individuals with disabilities, and veterans, among others.2 Many of the 
populations targeted by TRIO generally experience lower levels of college 
attendance and postsecondary degree achievement than their peers, 
which lessens their chances of higher lifetime earnings. 


The Higher Education Act Amendments of 1968 consolidated a trio of 
programs, Upward Bound, Talent Search, and the program that is now 
called Student Support Services, into TRIO.2 TRIO expanded to include 
five additional programs between 1972 and 1990, and there are now eight 
programs still known collectively as TRIO. Each program serves a 
specific population at different stages of educational pursuit, including 
middle school, high school, college, and graduate school. Education 
awards grants in each program through competitive processes based on 
the applicants’ capacity, plans, and prior experience operating TRIO 


1See 20 U.S.C. §§ 1070a-11 to 1070a-18. 


2For purposes of the TRIO programs, low-income individuals are those from a family 

whose taxable income in the calendar year prior to their initial participation in the program 
did not exceed 150 percent of the family income levels established by the Census Bureau 
for determining poverty status. Additionally, first-generation college students generally are 
those (1) whose parents did not complete a bachelor’s degree or (2) who regularly resided 
with and received support from only one parent who did not complete a bachelor’s degree. 


3See Pub. L. No. 90-575, § 105, 82 Stat. 1014, 1018-19. 
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programs.‘ While these programs have existed for at least 30 years, there 
has been recent interest in knowing more about their effectiveness. 


You asked us to review a broad range of issues related to the Federal 
TRIO programs, including how Education scores grant applications and 
assesses TRIO programs. This report examines: (1) how the additional 
points Education awards to returning TRIO grantees based on prior 
experience affect which applicants receive grants; (2) the types of 
colleges and other entities that receive these grants, and the extent to 
which colleges receiving grants enroll disadvantaged students; and (3) 
the extent to which Education assesses the results of the TRIO grant 
programs. 


To address all three objectives, we reviewed relevant federal laws and 
regulations, including the Higher Education Act of 1965, as amended, 
(HEA).5 We also interviewed Education officials, representatives from a 
policy research organization who have studied TRIO programs, and 
officials from an organization representing TRIO grantees. 


To examine the additional points Education awards to returning TRIO 
grantees based on prior experience, we analyzed Education’s applicant 
data for grant competitions from fiscal years 2015 through 2017—the 
most recent competitions at the time we began our review. We assessed 
the reliability of these data by examining them for outliers or potential 
errors, reviewing related documentation, and interviewing knowledgeable 
Education officials about how they collect these data. Based on these 
efforts, we found these data to be sufficiently reliable for the purposes of 
reporting how these points affect which applicants receive grants. We 
examined the total scores for all applicants, including experience points 
awarded to returning grantees. In order to determine how experience 
points may have affected the number of returning grantees that won new 
grants, we excluded these points from total scores and ranked all of the 


4The Higher Education Act of 1965, as amended, (HEA) generally requires Education to 
consider prior experience in making grants under TRIO. See 20 U.S.C. § 1070a-11(c)(2). 


5See Pub. L. No. 89-329, 79 Stat. 1219 (codified as amended at 20 U.S.C. §§ 1001 et 
seq.). 


STRIO grant competitions are staggered, and most competitions take place every 5 years. 
We excluded the Training Program for Federal TRIO Programs from our analyses 
because this program does not directly serve students. It provides grants to colleges and 
nonprofits to provide training and professional development for project directors and staff 
employed in TRIO programs. 
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applicants.” We also interviewed Education officials to learn about how 
they score applications. 


To describe the types of colleges and other entities that receive TRIO 
grants, we analyzed Education data regarding TRIO grantees that 
received funding in fiscal year 2018—the most recent data available at 
the time we began our review. To assess the reliability of these data, we 
interviewed knowledgeable Education officials and tested the data for 
errors. We found these data to be sufficiently reliable for the purposes of 
reporting the number of TRIO grantees and types of entities that receive 
TRIO grants. To assess the extent to which colleges receiving grants 
enroll disadvantaged students, we also used data from Education’s 
College Scorecard for academic year 2017-2018, the year that 
corresponded with Education’s most recent data on TRIO grantees. The 
College Scorecard is a database that combines information on colleges 
from several Education sources including the Free Application for Federal 
Student Aid, the National Student Loan Data System, and the Integrated 
Postsecondary Education Data System. Specifically, we analyzed data on 
the overall population of students at colleges that receive TRIO grants, 
including the overall proportion of students who receive Pell Grants 
(need-based federal grants to undergraduate students) and first- 
generation students. For this analysis, we examined the demographics of 
all students enrolled in colleges that receive TRIO grants, not just 
students who received TRIO services at these colleges. We reviewed 
documentation about how data are collected and tested the underlying 
data for outliers. We found these data to be sufficiently reliable for the 
purposes of reporting the proportion of students who receive Pell Grants 
and first-generation students. 


Although some TRIO programs also target students with disabilities and 
veterans, detailed data were not available for a high proportion of 
colleges nationwide on the number of these students they enroll. 
Therefore, we did not analyze the proportion of students with these 
characteristics enrolled at colleges that receive TRIO grants. We did 
include data on the characteristics of the small number of public school 
districts that received TRIO grants to provide context about additional 


7Education ranks applicants by total scores and awards grants until the available funding 
is exhausted. 
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grantees.’ However, we were unable to present comparable data on the 
characteristics of the nonprofit and other organizations that receive TRIO 
grants because similar information about these organizations is not 
available. 


To examine the extent to which Education assesses TRIO program 
results, we reviewed Education’s program goals and performance 
metrics. We also reviewed assessments of TRIO programs published 
over the past 15 years and the three most recent years of annual 
performance reports. We compared Education’s efforts to assess the 
results of TRIO programs against leading practices for performance and 
program assessments, as well as standards for project management. 


We conducted this performance audit from June 2019 to December 2020 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 


Eee 


Background 


Overview of Federal TRIO 
Programs 


TRIO provides services for individuals from disadvantaged backgrounds 
so they can prepare for, enroll in, and graduate from college or graduate 
school. TRIO grantees can include colleges, school districts, and 
community organizations, among others. Seven of the programs serve 
students through these grantees, while the eighth is a training program for 
directors and staff of TRIO projects (see table 1). Grantees provide 
required services, which may include academic tutoring or course 
selection advice, depending on the program. Grantees can also provide a 


8We used the American Community Survey, an annual survey conducted by the Census 
Bureau, to provide information on the number of students in these school districts whose 
parents had less than a bachelor’s degree. Once in college, these students are 
considered first-generation college students by Education. We used data from Education’s 
Office of Civil Rights to obtain information on the number of students in these school 
districts who were eligible to receive free and reduced-price lunch, an indicator of low- 
income students. 


9See Project Management Institute, Inc., A Guide to the Project Management Body of 


Knowledge (PMBOK® Guide), Sixth Edition (2017). PMBOK is a trademark of Project 
Management Institute, Inc. 
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range of permissible services to program participants, including 
mentorships, counseling, and exposure to higher education. 


Ee 
Table 1: Overview of Federal TRIO Programs 


Number of grantees and 


Program name (year founded) General purpose? participants (fiscal year 2018) 
Upward Bound (1964) College preparation for high school students 967 grantees 
70,852 participants 
Talent Search (1965) High school completion and college enrollment 473 grantees 
assistance for individuals who are age 11-27 309,905 participants 
Student Support Services (1968) Supports college students in completing their 1,069 grantees 
degree 202,795 participants 
Educational Opportunity Centers (1972) College enrollment assistance for adult students 140 grantees 
193,530 participants 
Veterans Upward Bound (1972) Helps military veterans prepare for college 62 grantees 
8,157 participants 
Training Program for Federal TRIO Provides professional development for TRIO staff 12 grantees 
Programs (1976) Does not directly serve students 
Ronald E. McNair Postbaccalaureate Helps students prepare for graduate school 187 grantees 
Achievement (1 986) 5,242 participants 
Upward Bound Math and Science (1990) Prepares high school students for college 212 grantees 
education programs and careers in math and 13,184 participants 
science 


Source: GAO analysis of Department of Education documents. | GAO-21-5 


*The TRIO programs that serve students generally require that at least two-thirds of those 
participating in the program be low-income individuals that are or would be first-generation college 
students, in addition to serving certain other populations. The exception is Student Support Services, 
which requires at least two-thirds of those participating be low-income individuals who are first- 
generation college students or individuals with disabilities; and the remaining participants must be 
low-income individuals, first generation college students, or individuals with disabilities. In addition, at 
least one-third of the individuals with disabilities served must also be low-income individuals. 


The CARES Act, passed in the wake of Coronavirus Disease 2019, 
allowed the Secretary of Education to modify the allowable uses for TRIO 
grants until September 30, 2020.19 For example, Education allowed 
grantees to provide computers or other technology to students to enable 
them to continue learning. Grantees could also receive approval to 
provide stipends to TRIO participants 18 years or older. As of August 7, 
2020, Education granted all requests (more than 260) for modifications 
under the CARES Act. 


10See Pub. L. No. 116-136, § 3518, 134 Stat. 281, 409. 
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Award Process 


Education holds grant competitions every 5 years for the seven TRIO 
programs that serve students and awards these grants for a period of 5 
years.11 Education can cancel grants during this period if grantees fail to 
meet certain annual performance targets, such as by serving fewer 
students than proposed in their applications. The competitions are 
staggered, so not all programs hold a competition during the same year. 
Education awards new grants based on scores derived from applications 
and the performance of returning grantees. 


e Application score. Each application receives an application score, 
which is the average of scores awarded by three peer reviewers. '2 
These reviewers score applications based on numerous criteria that 
are mostly specified in program regulations. ‘1? For example, in most 
TRIO competitions, applicants can receive up to 24 points for 
demonstrating the need for a project in the proposed community. 
Additionally, peer reviewers award applicants points based on the 
extent to which their performance targets are attainable and 
ambitious. During competitions that took place from fiscal year 2015 
through 2017, the maximum possible application score ranged from 
103 to 110 points, for the seven TRIO programs we analyzed. "4 


e Experience points. Education awards up to 15 additional points to 
returning grantees based on the extent to which they met 
performance targets specified in their most recent grant application. "5 
The level of consideration given to prior experience (i.e., the possible 
number of experience points) and the performance metrics Education 


11Grants for the Training Program for Federal TRIO Programs are awarded for a period of 
2 years, and Education holds a grant competition for this program every 2 years. 


12Education refers to this score as “the average reader score.” For the purposes of this 
report, we refer to this score as the application score. 


13See 34 C.F.R. pts. 643-647. Under TRIO program regulations, the maximum score for 
all criteria is generally 100 points. However, Education awards additional points 
(competitive preference) for TRIO applications that meet established annual priorities. 
Education describes these priorities in announcements for each grant competition. 


14We excluded the Training Program for Federal TRIO Programs from our analyses 
because it does not directly serve students. This program has a maximum application 
score of 75 points. 


15Generally under the HEA, Education is required to consider prior experience in making 
grants under TRIO. See 20 U.S.C. § 1070a-11(c)(2). Education refers to points awarded 
based on prior experience as “prior experience points.” They are specific to each 
grantee’s performance in a particular program and do not transfer across programs. For 
the purposes of this report, we refer to these points as experience points. 
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uses to calculate these points are set in statute.1® As part of their 
grant application, grantees determine their own performance target for 
each metric.'” Education uses the results that grantees submit in their 
annual performance reports to calculate how many experience points 
to award returning grantees during the next grant competitions. 


Education awards grants based on an applicant’s total score, which is the 
sum of the application score and experience points.18 For applicants who 
do not receive experience points, their application score and total score 
are the same (see fig. 1).19 Education ranks applicants by total scores 
and awards grants until the available funding is exhausted. 2° 


16See 20 U.S.C. § 1070a-11(c)(2)(A) & (f). Additionally, the Office of Management and 
Budget encourages agencies to consider the past performance of grantees when 
awarding new grants. Awarding grants to recipients with prior experience may provide 
assurance that future grants will achieve similar results. See Office of Management and 
Budget, Managing for Results: The Performance Management Playbook for Federal 
Awarding Agencies (Washington, D.C.: Apr. 27, 2020). 


17Experience points are based on the performance targets grantees set in the previous 
grant cycle as part of their application. 


18See 20 U.S.C. § 1070a-11(c)(2)(A) & (f). Additionally, the HEA requires Education to 
award TRIO grants in the order of the scores received by the application for such a grant 
in the peer review process and adjusted for prior experience. See 20 U.S.C. § 1070a- 
11(c)(3)(A). Furthermore, the HEA also requires Education to consider the needs of 
eligible participants in the area, institution of higher education or secondary school to be 
served. See 20 U.S.C. § 1070a-11(c)(2)(B). 


18The Training Program for Federal TRIO Programs only considers experience points 
when there is a tie in application scores and there is not enough funding for each applicant 
with that same application score. In those instances, Education adds experience points to 
determine an adjusted total score for those applications. As previously noted, we excluded 
this program from our analyses because it does not directly serve students. 


201f two or more applicants receive the same total score and there is not enough funding to 
award them all with grants, Education will generally choose among the tied applicants to 
serve geographic areas and eligible populations that have been underserved by the 
program. Statutory provisions establish a minimum grant award for the student-serving 
TRIO programs, unless the applicant requests a smaller amount. See 20 U.S.C. § 1070a- 
11(b)(3). 
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Figure 1: Example of Scoring Components of Federal TRIO Program Grant Competitions, Fiscal Year 2017 Upward Bound 
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Source: GAO analysis of Department of Education (Education) documents. | GAO-21-5 


(+) 15 maximum points 


Education awards returning grantees 
up to 15 additional points based on 
the extent to which they met the 
performance targets the grantees set 
during the previous grant cycle. 


118 maximum points 


The total score is the sum of the 
application score and experience 
points. 


*Under TRIO program regulations, the maximum score for all criteria is generally 100 points. 
However, Education awards additional points (competitive preference) for TRIO applications that 
meet established annual priorities. Education describes these priorities in announcements for each 
grant competition. The points that applicants receive for meeting these annual priorities are part the 
application score. Experience points are subsequently added to the application score. 
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percent of new grant awards during this period.21 Education’s longest- 


running TRIO programs, Upward Bound, Talent Search, and Student 
Support Services, awarded the highest percentages of grants to returning 
grantees (see fig. 2). 


Figure 2: Percentage of Grants Awarded to Returning Grantees in Most Recent Federal TRIO Program Grant Competitions, 
Fiscal Years 2015-2017 
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TRIO program (year program created) 
Source: GAO analysis of TRIO grant competition data from the Department of Education. | GAO-21-5 


Notes: GAO excluded the Training Program for Federal TRIO Programs from this analysis because it 
does not directly serve students. 


The Department of Education holds grant competitions every 5 years for 
these seven TRIO programs and awards these grants for a period of 5 
years. The competitions are staggered, so not all programs hold a 
competition during the same year. 


21During this time, there were 5,178 applicants, of which 2,665 were returning grantees. 
Returning grantees won 2,479 out of 3,121 new grant awards. Organizations can apply for 
and receive multiple grants within one program, and also across programs. 
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About 20 Percent of 
Returning Grantees Were 
Awarded New Grants as a 
Result of Receiving 
Experience Points 


Returning grantees were also more likely to win grants compared to new 
applicants. Specifically, 93 percent of returning grantees won grants 
compared to 26 percent of new applicants. 


Under the HEA, Education is generally required to consider prior 
experience in making grants under TRIO.22 According to our analysis of 
new grant awards from fiscal years 2015 through 2017, 23 percent of 
applicants that were awarded grants and received experience points won 
these awards over new applicants that had higher application scores.23 
When we ranked applicants based solely on their application scores (i.e., 
without experience points), we found that fewer returning grantees would 
have won awards.4 Specifically, if the most recent grant competitions did 
not factor prior experience into scores, the number of new grantees 
participating in the program would have almost doubled (see fig. 3). 


22See 20 U.S.C. § 1070a-11(c)(2). 


23The remaining 77 percent of grantees that received experience points had high enough 
application scores based on the other criteria that they could have won awards if total 
scores excluded experience points. 


24The number of awards depends on the funding each grantee receives. As part of its 
competition announcement, Education estimates the number of grants it will award based 
on the average dollar amount it expects each grantee to receive. Our analysis assumes 
the number of grant awards would remain the same. However, in an actual competition 
the number would change slightly depending on the funding each grantee receives. 
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Figure 3: Distributions of Most Recent Federal TRIO Program Grant Competition 
Outcomes and Estimated Outcomes without Experience Points, Fiscal Years 2015- 
2017 


Actual distribution of recent Estimated distribution of recent TRIO 
TRIO grant awards grant awards if scores excluded 
experience points 
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39% 61% 
New Returning 
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Source: GAO analysis of TRIO grant competition data from the Department of Education. | GAO-21-5 


Notes: Generally, under the Higher Education Act of 1965, as amended, Education is required to 
consider prior experience in making grants under TRIO. See 20 U.S.C. § 1070a-11(c)(2). 


The number of awards depends on the funding given to each grantee. For this estimate, GAO 
assumed the number of grant awards would remain the same, when in reality, the number would 
change slightly depending on the funding given to each grantee. 


GAO excluded the Training Program for Federal TRIO Programs from this analysis because it does 
not directly serve students. 


The Department of Education holds grant competitions every 5 years for the seven TRIO programs 
that serve students and awards these grants for a period of 5 years. The competitions are staggered, 
so not all programs hold a competition during the same year. 


Education awarded most returning grantees a high number of the 15 
possible experience points based on their performance in past years. For 
example, Education awarded 92 percent of returning grantees at least 10 
experience points (see fig. 4). According to Education officials, it is 
possible for returning grantees not to receive experience points if they fail 
to meet the performance targets specified in the grantees’ previous 
applications. We found that only one out of the 2,665 returning grantees 
that applied for new grants in fiscal years 2015 through 2017 did not 
receive any experience points. 
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Figure 4: Experience Points Returning Grantees Received in Most Recent Federal TRIO Program Grant Competitions, Fiscal 
Years 2015-2017 
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Source: GAO analysis of TRIO grant competition data from the Department of Education. | GAO-21-5 


Notes: GAO excluded the Training Program for Federal TRIO Programs from this analysis because it 
does not directly serve students. 


The percentages in this figure do not add to 100 due to rounding. 


The Department of Education holds grant competitions every 5 years for the seven TRIO programs 
that serve students and awards these grants for a period of 5 years. The competitions are staggered, 
so not all programs hold a competition during the same year. 


The only new applicants that won awards during the most recent grant 
competitions were those with nearly perfect application scores. For 
example, in the Upward Bound Math and Science program, new 
applicants needed a perfect application score to receive a grant. Similarly, 
new applicants in other programs had to score within one-third of a point 
to three points of a perfect score to receive a grant. In contrast, some 
returning grantees won new grants despite having lower application 
scores because they received experience points, and thus had higher 
total scores (see fig.5). 
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Figure 5: Lowest Application Score of Grant Winners in Most Recent Federal TRIO Program Grant Competitions, Fiscal Years 
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Source: GAO analysis of TRIO grant competition data from the Department of Education (Education). | GAO-21-5 


Note: This figure only shows application scores, which reflect the average of scores awarded by three 
peer reviewers. Under TRIO program regulations, the maximum score for all criteria is generally 100 
points. Additionally, Education awards points for TRIO applications that meet established annual 
priorities. The points that applicants receive for meeting these annual priorities are part the 
application score and thus, included in this figure. 


This figure does not include the additional points Education awards to TRIO grantees based on prior 
experience (i.e., experience points). Education uses total scores, which include both application 
scores and experience points, to award grants 


GAO excluded the Training Program for Federal TRIO Programs from this analysis because it does 
not directly serve students. 


The Department of Education holds grant competitions every 5 years for these seven TRIO programs 
and awards these grants for a period of 5 years. The competitions are staggered, so not all programs 
hold a competition during the same year. 


Education officials told us that experience points have benefits and 
drawbacks. Specifically, they noted experience points enable continuity of 
services for students receiving services through TRIO grants, but the 
points also reduce the number of new grantees that are awarded grants. 
Our analysis confirms that returning grantees have participated in TRIO 
over the course of many years. For example, a majority of returning 
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grantees (74 percent) received their first grant at least a decade ago and 
42 percent received their first grant at least 20 years ago.25 


Stakeholders we interviewed differed on their views of the value of 
experience points. Representatives from a policy research organization 
who have studied TRIO programs told us that experience points might 
prevent new applicants with promising proposals from winning awards. In 
contrast, officials we spoke to at the organization representing TRIO 
grantees said experience points are important because they help stabilize 
funding for supportive services that low-income students need in order to 
persist in college. These officials also noted that, since the goal of TRIO 
programs is to help disadvantaged students overcome nonfinancial 
obstacles to higher education, steady funding streams can help ensure 
these students have consistent access to necessary services. 


Public Colleges Are 


the Primary 


Recipients of TRIO 


Grants and 


Participating Colleges 


Enroll Varying 

Proportions of 
Disadvantaged 
Students 


The Majority of TRIO 
Grantees Are Public 
Colleges 


In fiscal year 2018, 92 percent of TRIO grantees were colleges, according 
to our analysis of Education data. The majority of these colleges were 
public institutions. Nonprofit organizations, school districts, and other 
organizations, such as those that provide services to school districts, also 
received TRIO grants (see fig. 6). 


25These data reflect the first year an organization won a grant in a particular program, and 
do not indicate that each grantee held their grant continuously during that time. 
Additionally, according to Education officials, Education’s electronic record files begin in 
1993. Therefore, it is unknown what year some organizations first received a grant. 
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Figure 6: Federal TRIO Program Grantees by Type, Fiscal Year 2018 
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Source: GAO analysis of fiscal year 2018 TRIO grantee data from the Department of Education. | GAO-21-5 


Notes: GAO categorized college grantees based on the highest undergraduate degree they offer: 2- 
year colleges issue associate’s degrees, and 4-year colleges issue bachelor’s degrees. Additionally, 
GAO excluded the Training Program for Federal TRIO Programs from this analysis because it does 
not directly serve students. 


“Other colleges are generally private nonprofit 2-year colleges except for one grantee that is a for- 
profit college. 


Colleges That Receive TRIO programs generally serve disadvantaged students—including low- 
TRIO Grants Enroll income students and first-generation students, among others—but 
: : colleges that receive TRIO grants enroll varying proportions of these 
adhe iat Sie t students in their overall student population. This is in part because 
isadvantage udents program participants do not necessarily enroll in the colleges from which 
they receive TRIO services.26 For example, Talent Search grantees, 


26Two exceptions are the Ronald E. McNair Postbaccalaureate Achievement and Student 
Support Services programs. Participants in these programs must be enrolled or accepted 
for enrollment at the college that provides them TRIO services. 
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many of which are colleges, provide services to participants as young as 
11 to help them complete high school and enroll in the college of their 
choice. Further, TRIO college grantees are not required as part of their 
participation in TRIO to enroll certain proportions of disadvantaged 
students in their general student populations.2” 


Our analysis of Education data on students receiving Pell Grants shows 
that colleges that receive TRIO grants enroll varying proportions of low- 
income students (see fig. 7). Pell Grants provide need-based federal 
financial aid to low-income undergraduate students. The proportion of 
students who received Pell Grants ranged from 9 percent to 98 percent in 
fiscal year 2018 at colleges that receive TRIO grants. The median 
proportion of students who received Pell Grants at these colleges was 35 
percent, which is below the national median of 46 percent of students 
enrolled in colleges overall. 


27In certain programs, such as Student Support Services, Education uses the proportion 
of disadvantaged students at the institution as one of the criteria to assess applicants. 
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Figure 7: Proportion of Pell Grant Recipients Enrolled at Colleges That Receive 
Federal TRIO Program Grants Compared to All Colleges, Fiscal Year 2018 


Percent 


100 


90 
83% 


80 


70 


59% 
60 


50 


40 
32% 
30 


20 


0, 
6 9% 10% 8% 


<20% of students 20% to 60% of students >60% of students 
receive Pell Grants receive Pell Grants receive Pell Grants 


aes Colleges that receive TRIO grants 


= All colleges 


Source: GAO analysis of College Scorecard and fiscal year 2018 TRIO grantee data from the Department of Education. | GAO-21-5 


Notes: GAO excluded the Training Program for Federal TRIO Programs from this analysis because it 
does not directly serve students. 


The percentages in this figure do not add to 100 due to rounding. 


Similarly, colleges that receive TRIO grants enroll varying proportions of 
first-generation students (see fig. 8). The proportion of students who were 
first-generation ranged from 10 percent to 70 percent in fiscal year 2018 
at these colleges. The median proportion of first-generation students 
enrolled in these colleges was 41 percent, which is below the national 
median of 48 percent of students enrolled in colleges overall. 
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Figure 8: Proportion of First-Generation Students Enrolled at Colleges That Receive 
Federal TRIO Program Grants Compared to All Colleges, Fiscal Year 2018 
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Source: GAO analysis of College Scorecard and fiscal year 2018 TRIO grantee data from the Department of Education. | GAO-21-5 


Notes: GAO excluded the Training Program for Federal TRIO Programs from this analysis because it 
does not directly serve students. 


The percentages in this figure do not add to 100 due to rounding. 


Approximately 11 percent of all colleges that receive TRIO grants have 
overall student populations that consist of less than 20 percent of 
students who receive Pell Grants or are first-generation students. Private 
nonprofit colleges that receive TRIO grants, on average, have lower 
proportions of Pell Grant recipients and first-generation students than 
public colleges that receive TRIO grants. 
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Characteristics of Public School Districts 
That Received TRIO Grants in Fiscal Year 
2018 


In fiscal year 2018, eight public school 
districts received TRIO grants. These school 
districts have moderate to high proportions of 
low-income and potential first-generation 
college students in their districts. In these 
districts the proportion of students eligible for 
free and reduced-price lunches, a proxy for 
low-income status, ranged from 34 percent to 
100 percent. Similarly, the proportion of 
students who are potential first-generation 
college students ranged from 54 percent to 97 
percent, for the seven school districts for 
which this information was available. 

Source: GAO analysis of American Community Survey data 


and data from the Department of Education’s Office for Civil 
Rights. | GAO-21-5 


Officials from the organization representing TRIO grantees we spoke to 
stated that a wide variety of colleges provide services to students through 
TRIO, including those that traditionally serve a greater proportion of 
higher-income students. Moreover, they said one of the purposes of TRIO 
is to help low-income, first-generation, and other disadvantaged students 
to prepare for and enroll in all types of colleges and universities. They 
added that Education encourages students participating in TRIO to apply 
to all types of schools, not just those with high concentrations of other 
low-income students. 28 


Education also awards TRIO grants to some public school districts, which 
serve moderate to high proportions of disadvantaged students (see 
sidebar). 


Education Measures 
TRIO Performance, 
but Has Taken Few 
Steps to Verify 
Results or Assess 
Effectiveness of 
Some Programs 


28They also noted that low-income students may improve their chances of graduation by 
attending more competitive institutions even if these colleges do not enroll high 
percentages of low-income students. 
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Education Uses Program 
Goals and Performance 
Metrics, but Has Taken 
Few Steps to Ensure the 
Reliability of Results 
Grantees Report 


Education uses program goals and performance metrics to measure 
TRIO performance.29 The overall goal for TRIO is to increase the 
percentage of low-income, first-generation college students who pursue 
higher education and complete college. Each program also has specific 
goals. For instance, a goal of Student Support Services, which had the 
most TRIO grantees in fiscal year 2018, is to increase the college 
retention and completion (i.e., graduation) rates of its participants. 
Student Support Services grantees measure progress toward that 
program’s goal by reporting the number of students who received 
services, along with the percentage of first-time college students who 
participated in the program and graduated from their college on time, 
among other metrics. Similarly, other programs also have performance 
metrics that track progress toward their goals. Education annually submits 
a report to Congress that documents the performance metrics of all TRIO 
programs, as required under the HEA.3° 


To measure TRIO’s program performance, Education uses individual 
reports from thousands of grantees. Education annually collects data from 
each TRIO grantee on the extent to which it met its targets related to its 
performance metrics.31 Education provides guidance to grantees on 
submitting results in their annual performance reports. Specifically, 
Education instructs grantees to report on the eligibility of program 
participants (e.g., if they are first-generation, low-income students), the 
types of services provided, and the educational attainment outcomes for 
participants. Education uses these data in several ways: (1) to measure 
progress toward program goals; (2) to make decisions about annually 


29We have previously reported that these are key components of effective program 
performance assessment. Program goals communicate what the agency proposes to 
accomplish and performance metrics permit the measurement of progress made toward 
the agency’s goals. For more information on performance assessment, see GAO, 
Performance Measurement and Evaluation: Definitions and Relationships, 
GAO-11-646SP (Washington, D.C.: May 2, 2011). 


30See 20 U.S.C. § 1070a-18(a). 


31During the application process, grantees set their own targets for these performance 
metrics, such as the number of participants who enroll in college. Education’s peer 
reviewers rate applicants based on the extent to which their performance targets are 
attainable and ambitious. 
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continuing grants32; and (3) as the basis of the experience points 
grantees receive when applying for a new grant. 


Although Education uses these data for decision-making, there is a risk 
that the data may not always be reliable. Education evaluates grantees 
using self-reported data about the extent to which they met their 
performance targets, and our analysis found that grantees who receive 
experience points for meeting their targets are more likely to win new 
grants. Therefore, grantees may have structural incentives to report 
results that would increase the likelihood of receiving a future award. This 
raises concerns about the reliability of the data grantees submit regarding 
the extent to which they met their performance targets. For example, 
officials from the organization representing TRIO grantees told us that 
with over 3,000 grantees reporting annually, there is a risk that some 
grantees may report inaccurate information.33 


Education does not routinely review the data grantees self-report about 
their results to help ensure their accuracy. While grantees are required to 
sign a statement attesting that the information they provide is true under 
the penalty of the law, there are few quality checks on this information. 
Education officials stated that they monitor grantee results for large 
fluctuations in reported performance from year to year or if a grantee 
does not meet its performance targets.34 However, the vast majority of 
grantees report meeting their performance targets. Focusing monitoring 
efforts on grantees that do not meet their targets limits Education’s ability 
to detect problems grantees do not self-report. For example, if a grantee 
were to incorrectly report meeting its performance targets without large 
fluctuations in performance, this would likely not be detected by 
Education’s current monitoring. Education officials also stated that the 


32After each year during the 5-year grant period, Education uses these performance 
reports to determine if each grantee should continue to receive the grant for the next year. 
Education officials told us that they have canceled very few grants during their 5-year 
grant period. 


33This organization did not identify any specific instances of grantees reporting inaccurate 
results. Our analysis focused on Education’s monitoring of data reliability risks. We did not 
review individual grantee data for accuracy. 


34Education officials stated that they also monitor grantees’ programmatic information and 


financial data for compliance with program requirements and to identify grantees that may 
need technical assistance. 
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agency performs some data checks when conducting site visits. 35 
However, Education officials also noted that these site visits occur 
infrequently due to limited staff and resources. From 2015 through 2019, 
Education officials stated that the agency conducted an average of 12 site 
visits per year for about 3,000 TRIO grantees. 


Since Education does not routinely review the data grantees report about 
their results to help ensure their accuracy, the agency may be missing 
opportunities to more accurately measure TRIO performance. Education’s 
Office of Inspector General previously highlighted broad concerns about 
the accuracy of the data the agency collects from grantees. In its fiscal 
year 2020 report on top management challenges, the Office of Inspector 
General noted concerns about Education’s controls over the accuracy 
and reliability of program performance data grantees provided across the 
agency’s programs.%6 The report stated that effective monitoring and 
oversight are essential in ensuring grantees meet requirements and 
achieve program goals. Education officials acknowledged that there is a 
risk of TRIO grantees reporting errors in their performance data. We have 
previously reported that a leading practice for assessing grant programs 
is that agencies should examine grantee performance data for quality and 
reliability. Examining data reliability could include spot-checking grantee 
data. It could also include asking questions to grantees and requesting 
documentation from them about how they collect data and ensure data 
reliability, and then reviewing these responses and documents. 3” Taking 
additional steps to review the accuracy of results TRIO grantees submit 
could help Education ensure that it provides experience points only to 
returning grantees that have actually met performance targets. It would 


35Education officials stated that site visits can be triggered by a complaint of potential 
fraud, a grantee request for technical assistance, or random selection, among other 
reasons. They stated that site visits typically involve reviewing a random sample of 
student records and activity logs to monitor grantee compliance for requirements 
pertaining to student eligibility, services provided, and educational progress, among other 
things. They added that these limited data checks have rarely identified cases involving 
improper or fraudulent activities. 


36Department of Education, Office of Inspector General, FY 2020 Management 
Challenges Facing the U.S. Department of Education (November 2019). 


37GAO, Native American Youth: Agencies Incorporated Almost All Leading Practices 
When Assessing Grant Programs That Could Prevent or Address Delinquency, 
GAO-20-600 (Washington, D.C.: Aug. 6, 2020). 
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Education Has Studied 
Some TRIO Programs, but 
Has Gaps in Its Evidence 
on Program Effectiveness 


also help Education better track progress toward meeting its goals and 
ensure it provides reliable information in its annual reports to Congress. 


Education has studied some TRIO programs and program activities, but 
has gaps in its evidence on the effectiveness of TRIO.38 Education has 
conducted some assessments to identify practices that further the 
outcome goals of TRIO (as required under the HEA), describe program 
performance, and examine the overall effectiveness of several individual 
TRIO programs.%? While Education’s performance metrics monitor 
progress against each TRIO program’s goals, assessments of overall 
program effectiveness typically examine program performance more in- 
depth. These assessments typically examine a broader range of 
information on program performance and its context than is feasible to 
monitor on an ongoing basis. They also allow for an overall assessment 
of whether participation in the program leads to better outcomes and 
whether adjustments may improve results. 4° 


However, Education’s evidence regarding the effectiveness of TRIO is 
incomplete in part because: 


e Education has never assessed some programs for overall 
effectiveness. Education has assessed the overall program 
effectiveness of four of the seven TRIO programs that serve students. 
However, Education has never assessed the overall effectiveness of 
three programs: Veterans Upward Bound or Educational Opportunity 
Centers, each of which was founded in 1972, or Ronald E. McNair 


38Some of Education’s recent studies focus on program activities. For example, Education 
is currently studying the effect of sending personalized text messages to students in the 
Educational Opportunity Centers program to see if these messages increase the rates in 
which students complete the Free Application for Federal Student Aid form and enroll in 
college. Education expects to release this study in 2022. Studies of program activities 
inform Education about the effectiveness of certain practices; however, they do not 
provide Education or Congress information on the overall effectiveness of a given TRIO 
program. In addition, Education has also published descriptive profiles of some TRIO 
programs. Education officials stated that the agency plans to review, and potentially 
publish, further descriptive profiles that will present outcome data associated with program 
objectives. 


39The HEA requires Education to rigorously evaluate the effectiveness of TRIO programs 
for the primary purpose of identifying particular practices that further the achievement of 
program outcome criteria. See 20 U.S.C. § 1070a-18(b)(1)(A) & (2). 


40See GAO-11-646SP. 
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Postbaccalaureate Achievement, which was founded in 1986.41 Since 
each TRIO program has individual goals and targets different 
populations, an assessment of one TRIO program is not reflective of 
the other programs’ effectiveness. 


e Education’s studies of some programs have limited applicability 
to current participants. Education’s studies of two programs are 
outdated. For instance, Education’s most recent studies of the overall 
effectiveness of Upward Bound and of Upward Bound Math and 
Science were based on participants that received services in the mid- 
1990s, and there were changes to these programs since that time. 42 
Specifically, for the Upward Bound program, Education subsequently 
announced a new priority that limited which students grantees could 
target to participate.43 Additionally, the composition of Upward Bound 
Math and Science grantees has changed since these programs were 
studied.44 In light of these changes, these studies’ findings may not be 
directly applicable to current participants. 


The HEA currently requires Education to “rigorously evaluate” the 
effectiveness of TRIO programs for the primary purpose of identifying 
particular practices that further the achievement of program outcome 
criteria.45 The HEA also prohibits Education from requiring grantees to 
participate in a program study that would require them to deny eligible 


41While Education released a study on the Ronald E. McNair Postbaccalaureate 
Achievement program in 2008, that study did not assess the effectiveness of the program. 


42Education released the study of Upward Bound in 2009 and the study of Upward Bound 
Math and Science in 2010. See Department of Education, Office of Planning, Evaluation 
and Policy Development, Policy and Program Studies Service, The Impacts of Regular 
Upward Bound on Postsecondary Outcomes Seven to Nine Years After Scheduled High 
School Graduation (Washington, D.C.: 2009) and The Impacts of Upward Bound Math- 
Science on Postsecondary Outcomes 7-9 Years After Scheduled High School Graduation 
(Washington, D.C.: 2010). 


43See Upward Bound Program, 71 Fed. Reg. 55,447, 55,447-48 (Sept. 22, 2006). Authors 
of the 2009 Upward Bound study noted that this change could “substantially modify” the 
composition of Upward Bound participants. TRIO Program regulations were also amended 
in 2010. See High School Equivalency Program and College Assistance Migrant Program, 
The Federal TRIO Programs, and Gaining Early Awareness and Readiness for 
Undergraduate Program, 74 Fed. Reg. 65,712 (Oct. 26, 2010). 


44 At the time Upward Bound Math and Science participants were studied, nearly 90 
percent of grantees were 4-year colleges. Since that time, the proportion of 4-year 
colleges has decreased. 


45See 20 U.S.C. § 1070a-18(b)(1)(A) & (2). 
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students access to these programs.46 Education previously interpreted 
this prohibition as preventing it from performing randomized controlled 
trials as a method to assess these programs.4’ However, in November 
2020, Education noted that it had further reviewed this statutory provision 
and concluded that the HEA does not universally prohibit the most 
rigorous evaluations that use a randomly assigned control group (see 
app. 1).48 


Education may also be able to use various other methods to assess the 
overall effectiveness of TRIO programs. For example, Education released 
a study in 2019 that compared outcomes of program participants in the 
Student Support Services program to demographically similar students 
who did not participate in the program.49 Additionally, in our prior work, we 
have highlighted several types of rigorous assessment methods agencies 
can use if randomized controlled trials are not possible. These include (1) 
the quasi-experimental approach that Education used to evaluate the 
Student Support Services program (i.e., compare the outcomes of 
participants and non-participants with similar characteristics), (2) 
statistical analyses of observational data, or (3) in-depth case studies in 


46See 20 U.S.C. § 1070a-18(b)(3). 


47Education’s fiscal year 2021 budget justification included a legislative proposal to 
remove this statutory provision, which would allow Education to assess the overall 
program effectiveness of TRIO programs using the most rigorous methodological 
approaches. Education’s proposed changes would also allow it to conduct assessments 
for a broader range of purposes. 


48For example, Education stated that it could assess the effectiveness of a program in 
situations where the number of eligible applicants exceeds the number of funded slots for 
participating students. Education said in such a situation, a TRIO grantee could allocate 
the available slots among the eligible applicants on a random basis, and any denial of 
services for an eligible student would be the result funding limitations, not the result of the 
random selection. Education noted the effectiveness of the program or project could then 
be rigorously evaluated using a randomly selected control group, by comparing the 
students randomly selected to participate in the program with the students randomly 
selected to not participate in the program. 


49The study found Student Support Services participants who initially enrolled in 2-year 
colleges had significantly higher college completion rates than non-participants. However, 
participants who initially enrolled in 4-year colleges did not complete college at 
significantly different rates from non-participants. See Department of Education, Office of 
Postsecondary Education, Student Service, Comparing Student Outcomes Between 
Student Support Services Participants and Nonparticipants in the 2004/09 Beginning 
Postsecondary Students Longitudinal Study (Washington, D.C.: 2019). 
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some circumstances.5° Education could consider any of these methods to 
assess the effectiveness of other TRIO programs. Without assessing the 
programs that have never been evaluated or reviewed for overall 
effectiveness or the programs that have changed substantially since their 
most recent assessments, Education cannot determine the extent to 
which these TRIO programs are effectively serving participants. *" 


Education officials said they are in the early stages of planning future 
studies of the agency’s various programs, but did not provide any 
documentation of this effort. Additionally, as of August 2020, they had not 
yet decided which topics to explore, if these studies would include 
assessments of the overall effectiveness of TRIO programs, or milestone 
dates for conducting any such studies.52 Guidance from the Office of 
Management and Budget states that rigorous, independent program 
assessments can be a key resource in determining whether government 
programs are achieving their intended outcomes as well as possible and 
at the lowest possible cost.5° Assessing those TRIO programs that have 
not been previously assessed for overall effectiveness as well as those 
that have experienced programmatic changes since they were last 
reviewed would provide Education information on whether participation in 
TRIO programs leads to better student outcomes. Leading practices for 
assessments state that agencies should require periodic assessments of 
programs throughout their life because, as programs change over time, 
assessing them periodically can provide ongoing feedback and insights 
about the programs’ effectiveness. In addition, standards for project 


50GAO, Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective 
Interventions, GAO-10-30 (Washington, D.C.: Nov. 23, 2009). 


51In May 2019, a public policy research organization raised concerns about the extent to 
which Education has assessed TRIO outcomes. It recommended that Education, among 
other things, assess the effectiveness of TRIO programs. 


52The Foundations for Evidence-Based Policymaking Act of 2018 provides Education an 
opportunity to plan future assessments of TRIO program effectiveness. Education is 
preparing to implement this act, which requires agencies to plan and organize agency- 
wide evidence-building activities by 2021. Education could leverage this agency-wide 
evidence-building effort to supplement the evidence it has on TRIO effectiveness. See 
Pub. L. No. 114-435, tit. |, 132 Stat. 5529, 5529-34 (2019) (codified at 5 U.S.C. §§ 311- 
315). 


53Office of Management and Budget, Increased Emphasis on Program Evaluations, M-10- 
01 (Washington, D.C.: October 2009). 


54American Evaluation Association, An Evaluation Roadmap for a More Effective 
Government, accessed July 23, 2020, https://www.eval.org/evaluationroadmap. 
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Conclusions 


Recommendations for 
Executive Action 


Agency Comments 
and Our Evaluation 


management state that managing a project involves developing a plan 
with specific actions and milestone dates.55 By developing such a plan for 
conducting future assessments of TRIO programs under its statutory 
authority, Education would be better positioned to assess the extent to 
which TRIO improves higher educational outcomes for disadvantaged 
students. 


Higher education is an important pathway to economic opportunities. 
Congress has provided over $1 billion annually for TRIO programs to 
support disadvantaged students in preparing for, applying to, and 
graduating from college. Although each TRIO program has performance 
metrics and goals, Education does not routinely take steps to verify the 
data grantees report. The integrity of these data are important because 
Education uses the information to track progress toward program goals 
and determine which applicants earn additional points based on past 
performance for the next grant competition. Additionally, gaps in 
Education’s assessments of TRIO programs impede its ability to 
determine the programs’ overall effectiveness. Taking additional steps to 
assess the reliability of grantees’ performance data and the overall 
effectiveness of the TRIO programs would better position Education to 
determine the extent to which TRIO improves higher educational 
outcomes for disadvantaged students. 


We are making the following two recommendations to Education: 


The Assistant Secretary for the Office of Postsecondary Education should 
take additional steps to ensure the performance data TRIO grantees 
report are reliable. (Recommendation 1) 


The Assistant Secretary for the Office of Postsecondary Education should 
develop a plan—with specific actions and milestone dates—for assessing 
the effectiveness of TRIO programs that serve students using methods 
that are consistent with its statutory authority. (Recommendation 2) 


We provided a draft of this report for review and comment to the 
Department of Education. In its comments, reproduced in appendix I, 
Education concurred in part with our first recommendation and in full with 


55See Project Management Institute, Inc. A Guide to the Project Management Body of 
Knowledge (PMBOK® Guide), Sixth Edition (2017). PMBOK is a trademark of Project 
Management Institute, Inc. 
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our second recommendation. Education also provided technical 
comments, which we incorporated as appropriate. 


Regarding our first recommendation on performance information 
submitted by grantees, Education agreed that this information is important 
and the department has an interest in improving its reliability. Education 
also noted a number of steps it could take to better monitor the reliability 
of the different types of performance data grantees submit. For example, 
a number of the TRIO programs require grantees to collect and report 
student level information on postsecondary enrollment, persistence, and 
degree attainment outcome data that Education could cross-check 
against existing internal and external data systems. For other programs, 
Education is considering identifying other methods for verifying 
performance data, including monitoring TRIO grantees’ programs and 
adding questions to grantees’ required annual reports regarding their 
methodology for compiling student level data. 


However, Education noted that it does not have access to underlying data 
sources that can independently verify certain data that grantees in some 
programs are required to submit, such as high school persistence, 
rigorous course enrollment, exposure to research opportunities, student 
grade point average, and secondary school completion. We recognize 
that Education may not have access to all of the underlying data. 
However, verifying information where possible is an important step toward 
improving oversight of grantees. In addition, as we stated in the report, 
Education could ask grantees to explain how they collect data and 
request related documentation. Education stated that it could require 
grantees to explain their methodology for compiling student level data. 
Such action could help identify data reliability concerns even in cases 
where independent verification is more difficult. 


Education also noted that its ability to verify data submitted by grantees is 
dependent on the availability of appropriate funds for staffing. However, 
many of the steps the department identified to better monitor data 
reliability may not necessarily require additional staffing costs, such as 
leveraging the department's existing internal data or requiring grantees to 
document their methodology. Further, Education could consider other 
ways to limit costs, such as through a risk-based approach of monitoring 
a sample of grantees’ performance data. Prioritizing staff resources 
toward oversight and monitoring is an essential component of program 
administration, particularly for programs like TRIO that rely on the results 
of prior performance to award future grants. Taking steps, such as those 
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identified in its written comments, could help Education ensure the 
reliability of TRIO grantee performance data. 


Regarding our second recommendation that Education develop a plan to 
evaluate programs, Education concurred and stated that it plans to 
evaluate TRIO programs using the most rigorous methods available given 
statutory limitations and will develop a plan with specific actions and 
milestone dates. Education also noted that it is mindful of the statutory 
limitations of the HEA, under which Education cannot require grantees to 
participate in an evaluation that requires TRIO grantees to recruit 
additional students beyond those the TRIO program or project would 
normally recruit or that results in the denial of services for an eligible 
student under the program or project. Education further stated that, upon 
further review of this prohibition, it concluded that the HEA does not 
universally prohibit the most rigorous evaluations that use a randomly 
assigned control group. Additionally, Education described the kinds of 
evaluations it could perform. We incorporated this information into the 
report, where appropriate. 


As agreed with your office, unless you publicly announce the contents of 
this report earlier, we plan no further distribution until 30 days from the 
report date. At that time, we will send copies to the appropriate 
congressional committees and the Secretary of Education. In addition, the 
report will be available at no charge on the GAO website at 
http:/Awww.gao.gov. 


If you or your staff have any questions concerning this report, please 
contact me at (617) 788-0534 or emreyarrasm@gao.gov. Contact points 
for our Offices of Congressional Relations and Public Affairs may be 
found on the last page of this report. GAO staff who made key 
contributions to this report are listed in appendix II. 


Sincerely yours, 


Welrae Evry. Qnrec 


Melissa Emrey-Arras, Director 
Education, Workforce, and Income Security Issues 
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UNITED STATES DEPARTMENT OF EDUCATION 
OFFICE OF POSTSECONDARY EDUCATION 


November 17, 2020 


Ms. Melissa Emrey-Arras 

Director, Education, Workforce, 

and Income Security Issues 

U.S. Government Accountability Office 
441 G Street, NW 

Washington, DC 20548 


Dear Ms. Emrey-Arras: 


Thank you for providing the U.S. Department of Education (Education) with the U.S. 
Government Accountability Office’s (GAO’s) draft report titled, “Higher Education: Department 
of Education Should Further Assess College Access Grant Programs” (GAO-21-5; Job Code 
103590). We appreciate GAO’s audit work and the opportunity to comment on the draft report. 
In the responses that follow, we acknowledge administrative and statutory constraints that 
prevent Education from fully implementing the recommendations. Nonetheless, we agree that 
we can continue to improve Education’s processes for ensuring grantees’ performance data 
reliability, and we will continue to identify evaluation strategies that have the potential to 
improve outcomes in the Federal TRIO Programs (TRIO). 


Recommendation 1: The Assistant Secretary for the Office of Postsecondary Education should 
take additional steps to ensure the performance data TRIO grantees report are reliable. 


Response: Education concurs in part with the Recommendation. We agree to the extent that we 
are always interested in improving the reliability of the performance data and recognize its 
importance. To do this, there are a number of complexities that will need to be addressed. We 
will need to tailor how we examine data reliability based on a number of factors such as the type 
of grant, type of data collected, and instrument used to collect the data. For a number of the 
TRIO programs, such as the Student Support Services, Upward Bound/Upward Bound Math and 
Science, and Ronald E. McNair Postbaccalaureate Achievement programs, which collect and 
report student level information, Education can access internal and other external data that can be 
used to evaluate the reliability of reports submitted by TRIO grantees on postsecondary 
enrollment, persistence, and degree attainment outcome data. While other available data include 
both internal data such as the National Student Loan Data System (NSLDS) and external data 
such as the National Student Clearinghouse (NSC), such databases are not complete. The Talent 


400 Maryland Avenue, 8.W., Washington, DC 20202 
www.ed. gov 


The Department of Education's mission is to promote student achievement and preparation for global Competitiveness 
by fostering educational excellence and ensuring equal access. 
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Search TRIO program’s reports of student rosters may offer an opportunity to perform similar 
data reliability reviews. 


For TRIO programs not mentioned above, Education is considering identifying other statistically 
appropriate methods for verifying performance data including potentially conducting monitoring 
of TRIO grantees’ programs. Education will consider a number of factors in selecting grantees 
to monitor. 


Our ability to conduct this data verification is dependent on the availability of appropriate funds 
for staffing. In addition, Education will consider reviewing the TRIO programs’ Annual 
Performance Report (APR) data collection instruments and, where appropriate, add questions 
regarding grant project methodology for compiling student level data. This information would 
help identify grantees with possible data reliability problems, although we believe it may be 
possible that implementing this data verification process change rigorously could require an 
increase to the current 2 of | percent cap on TRIO administrative expenses under 402A(g) of the 
Higher Education Act of 1965, as amended (HEA). 


The possible practices mentioned above have limitations or challenges. Foremost, these steps 
focus on a small set of program outcomes. However, program regulations specify other 
outcomes of interest that would not be captured within the NSLDS or other data resources. For 
example, some grant programs are concerned with high school persistence, rigorous course 
enrollment, exposure to research opportunities, student grade point average, and secondary 
school completion. Grant projects report these outcomes (and others) for various TRIO grant 
programs. We do not have access to underlying data sources that can independently verify APR 
reporting for metrics related to these outcomes, but we will consider ways to further verify the 
data. While we concur in considering improvements, based on the factors cited above the results 
may be limited. 


Recommendation 2: The Assistant Secretary for the Office of Postsecondary Education should 
develop a plan—with specific actions and milestone dates—for assessing the effectiveness of 
TRIO programs that serve students using methods that are consistent with its statutory authority. 


Response: Education concurs with this Recommendation because it is based on the statutory 
requirement to “rigorously evaluate the effectiveness of the [TRIO] programs and projects” 
under section 402H of the HEA, codified at 20 U.S.C. § 1070a—18(b)(1)(A). Education also 
concurs with the recommendation to develop a plan with specific actions that comply with the 
statutory mandate and milestone dates for those actions. 


Education is mindful of the statutory limitations in section 402H of the HEA, under which the 
Secretary cannot require grantees to participate in an evaluation that requires TRIO grantees to 
recruit additional students or that “results in the denial of services for an eligible student under 
the program or project.” However, upon further review of this prohibition, Education has 
concluded that it does not universally prohibit the most rigorous kinds of evaluations that use a 
randomly assigned control group. 
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For example, in situations where the number of eligible applicants exceeds the number of funded 
slots for participating students, a TRIO grantee could allocate the available slots among the 
eligible applicants on a random basis; in such a situation, any “denial of services for an eligible 
student under the program or project” would be the result of the limitations on funding, not the 
result of the random selection. The effectiveness of the program or project could then be 
rigorously evaluated using a randomly selected control group, by comparing the students 
randomly selected to participate in the program with the students randomly selected to not 
participate in the program. 


Likewise, Education interprets the statute to only prohibit an evaluation that results in a general 
“denial of services for an eligible student” under a TRIO program or project. As a result, an 
evaluation that randomly assigns some students in the program to receive a particular new 
practice as a treatment while assigning the rest of the students in the program to receive the same 
program services without that new practice—in order to rigorously evaluate the effectiveness of 
that practice to identify “particular practices that further the achievement of the outcome criteria’ 
dictated by the TRIO statute, 20 U.S.C. § 1070a—18(b)(2)(B)—would not result in a general 
“denial of services” for any eligible student, and would therefore be allowed under the statute. 


o 


In 2021 and 2022, IES plans to conclude rigorous, random assignment evaluations of two 
promising practices, one in the Upward Bound program and the other in the Educational 
Opportunity Centers program. Neither study would result in the “denial of services” to an 
eligible student. 


Of course, to the extent that in some instances the best methods of “rigorously evaluat[ing] the 
effectiveness of the [TRIO] programs and projects” are prohibited because they would result in 
“the denial of [TRIO] services” to one or more eligible students, Education will not conduct such 
analyses. Education will instead choose among the next most rigorous methods available to 
evaluate the effectiveness of the TRIO programs and projects at issue, while clearly stating the 
limitations of conclusions based on these less rigorous methods. Education continues to support 
the removal of the statutory limitations on evaluation, but will evaluate the TRIO programs as 
effectively as possible given statutory constraints. 


Thank you for the opportunity to provide a written response to the draft GAO report. We 
encourage GAO to consider the information provided in this response when preparing its final 
report. As indicated, we will use the information in this report to improve our TRIO program 
performance data reliability, as well as continue to explore strategies to evaluate the TRIO 
programs in accordance with current statutory authority. 


Sincerely, 


Digitally signed by Robert 


Robert King ae 2020.11.17 
11:00:33 -05'00' 
Robert L. King 
Assistant Secretary 
Office of Postsecondary Education 
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